Multi-task Learning Framework Research Articles

Conversion of single-energy computed tomography to parametric maps of dual-energy computed tomography using convolutional neural network.

We propose a deep learning (DL) multi-task learning framework using convolutional neural network (CNN) for a direct conversion of single-energy CT (SECT) to three different parametric maps of dual-energy CT (DECT): Virtual-monochromatic image (VMI), effective atomic number (EAN), and relative electron density (RED). We propose VMI-Net for conversion of SECT to 70, 120, and 200 keV VMIs. In addition, EAN-Net and RED-Net were also developed to convert SECT to EAN and RED. We trained and validated our model using 67 patients collected between 2019 and 2020. SECT images with 120 kVp acquired by the DECT (IQon spectral CT, Philips) were used as input, while the VMIs, EAN, and RED acquired by the same device were used as target. The performance of the DL framework was evaluated by absolute difference (AD) and relative difference (RD). The VMI-Net converted 120 kVp SECT to the VMIs with AD of 9.02 Hounsfield Unit, and RD of 0.41% compared to the ground truth VMIs. The ADs of the converted EAN and RED were 0.29 and 0.96, respectively, while the RDs were 1.99% and 0.50% for the converted EAN and RED, respectively. SECT images were directly converted to the three parametric maps of DECT (ie, VMIs, EAN, and RED). By using this model, one can generate the parametric information from SECT images without DECT device. Our model can help investigate the parametric information from SECT retrospectively. Deep learning framework enables converting SECT to various high-quality parametric maps of DECT.

Component Identification and Depth Estimation for Structural Images Based on Multi-Scale Task Interaction Network

Component identification and depth estimation are important for detecting the integrity of post-disaster structures. However, traditional manual methods might be time-consuming, labor-intensive, and influenced by subjective judgments of inspectors. Deep-learning-based image visual inspection is a new approach to overcome these problems, but repeated modeling is required for different inspection tasks, which limits inspection accuracy and practical deployment efficiency. In this study, it is observed that the matched ratios of pixel pairs between component identification and depth estimation reach a high value, which indicates the dual tasks are highly related. Therefore, the Multi-Scale Task Interaction Network (MTI-Net) is proposed for structural images to simultaneously accomplish both tasks for accurate and efficient structural inspection. It propagates distilled task information from lower to higher scales. Then, it aggregates the refined task features from all scales to produce the final per-task predictions. Experiments show that MTI-Net delivers the full potential of multi-task learning, with a smaller memory footprint and higher efficiency compared to single-task learning. For the evaluation metrics of model performance, the mean Intersection over Union (mIoU) of component identification improves by 2.30, and root mean square error (RMSE) drops by 0.36 m with the aid of the multi-task strategy. The multi-task deep learning framework has great potential value in engineering applications.

Probabilistic movement primitives based multi-task learning framework

With the increasing complexity of industrial production and manufacturing tasks, industrial robots are expected to learn intricate operations from simple actions easily and quickly with adaption to dynamic environment. In this paper, a task-parameterized multi-task learning framework is proposed to facilitate rapid learning of operational skills for industrial robots. In this framework, a conditional Probabilistic Movement Primitives (ProMP) is firstly employed to the single-task learning. Using the conditional probability calculation, the extrapolation issue in Learning from Demonstration (LfD) is addressed, enabling robots to learn beyond teaching. Subsequently, the single-task is extended to multi-task scenario by proposing a multi-task learning approach where each single task executes an extrapolation learning. The learned skill can meet the multiple task requirements through an iterative modulation manner. The effectiveness of the proposed framework is validated through both the simulation and a 7-DoF Franka-Emika robot experiment in a predefined task scenario. Furthermore, the outperformance of the proposed method is demonstrated by comparing with the state-of-art movement primitives based learning method.

A Comprehensive Framework for Face age Synthesis, Voice Transformation, and Cartoon Generation

This paper presents a comprehensive framework addressing age-invariant face recognition (AIFR), face age synthesis (FAS), voice transformation, and avatar generation. Traditional AIFR techniques focus on minimizing age-related variations in face recognition but lack visual interpretability, while FAS methods aim to synthesize faces of different ages but often compromise recognition due to artifacts. To address these limitations, we propose MTLFace, a multi-task learning framework that simultaneously handles AIFR and FAS tasks. MTLFace employs attention-based feature decomposition to separate identity and age-related features spatially, improving interpretability. Additionally, we introduce an identity conditional module for fine-grained face age synthesis, enhancing the naturalness of synthesized faces. Unlike conventional methods that achieve age group-level synthesis, our identity conditional module enables identity-level synthesis, resulting in smoother age transitions and preserving individual facial characteristics. Leveraging high-quality synthesized faces, we enhance AIFR performance via selective fine-tuning, where synthesized faces are used to augment training data, leading to improved robustness against age variations. Furthermore, we contribute a large cross-age face dataset with annotations, facilitating research in age-related tasks. In addition to AIFR and FAS, we explore limitations in existing methodologies for voice transformation and propose advancements to improve speech quality and address current challenges. Lastly, we introduce a novel approach for cartoon face generation, utilizing component-based facial feature extraction and template matching to produce diverse and stylistic cartoon faces. Experimental results across various benchmarks demonstrate the effectiveness of our approaches in achieving superior performance in face recognition, voice transformation, and cartoon face generation, thereby contributing to the advancement of multimodal synthesis technologies with potential applications in entertainment, virtual communication, and assistive technologies

MultiMatch: Multi-task Learning for Semi-supervised Domain Generalization

Domain generalization (DG) aims at learning a model on source domains to well generalize on the unseen target domain. Although it has achieved great success, most of the existing methods require the label information for all training samples in source domains, which is time-consuming and expensive in the real-world application. In this article, we resort to solving the semi-supervised domain generalization (SSDG) task, where there are a few label information in each source domain. To address the task, we first analyze the theory of multi-domain learning, which highlights that (1) mitigating the impact of domain gap and (2) exploiting all samples to train the model can effectively reduce the generalization error in each source domain so as to improve the quality of pseudo-labels. According to the analysis, we propose MultiMatch, i.e., extending FixMatch to the multi-task learning framework, producing the high-quality pseudo-label for SSDG. To be specific, we consider each training domain as a single task (i.e., local task) and combine all training domains together (i.e., global task) to train an extra task for the unseen test domain. In the multi-task framework, we utilize the independent batch normalization and classifier for each task, which can effectively alleviate the interference from different domains during pseudo-labeling. Also, most of the parameters in the framework are shared, which can be trained by all training samples sufficiently. Moreover, to further boost the pseudo-label accuracy and the model’s generalization, we fuse the predictions from the global task and local task during training and testing, respectively. A series of experiments validate the effectiveness of the proposed method, and it outperforms the existing semi-supervised methods and the SSDG method on several benchmark DG datasets.

Enhancing Transcription Factor Prediction through Multi-Task Learning (Student Abstract)

Transcription factors (TFs) play a fundamental role in gene regulation by selectively binding to specific DNA sequences. Understanding the nature and behavior of these TFs is essential for insights into gene regulation dynamics. In this study, we introduce a robust multi-task learning framework specifically tailored to harness both TF-specific annotations and TF-related domain annotations, thereby enhancing the accuracy of TF predictions. Notably, we incorporate cutting-edge language models that have recently garnered attention for their outstanding performance across various fields, particularly in biological computations like protein sequence modeling. Comparative experimental analysis with existing models, DeepTFactor and TFpredict, reveals that our multi-task learning framework achieves an accuracy exceeding 92% across four evaluation metrics on the TF prediction task, surpassing both competitors. Our work marks a significant leap in the domain of TF prediction, enriching our comprehension of gene regulatory mechanisms and paving the way for the discovery of novel regulatory motifs.

Continual Relation Extraction via Sequential Multi-Task Learning

To build continual relation extraction (CRE) models, those can adapt to an ever-growing ontology of relations, is a cornerstone information extraction task that serves in various dynamic real-world domains. To mitigate catastrophic forgetting in CRE, existing state-of-the-art approaches have effectively utilized rehearsal techniques from continual learning and achieved remarkable success. However, managing multiple objectives associated with memory-based rehearsal remains underexplored, often relying on simple summation and overlooking complex trade-offs. In this paper, we propose Continual Relation Extraction via Sequential Multi-task Learning (CREST), a novel CRE approach built upon a tailored Multi-task Learning framework for continual learning. CREST takes into consideration the disparity in the magnitudes of gradient signals of different objectives, thereby effectively handling the inherent difference between multi-task learning and continual learning. Through extensive experiments on multiple datasets, CREST demonstrates significant improvements in CRE performance as well as superiority over other state-of-the-art Multi-task Learning frameworks, offering a promising solution to the challenges of continual learning in this domain.

MCSSME: Multi-Task Contrastive Learning for Semi-supervised Singing Melody Extraction from Polyphonic Music

Singing melody extraction is an important task in the field of music information retrieval (MIR). The development of data-driven models for this task have achieved great successes. However, the existing models have two major limitations: firstly, most of the existing singing melody extraction models have formulated this task as a pixel-level prediction task. The lack of labeling data has limited the model for further improvements. Secondly, the generalization of the existing models are prone to be disturbed by the music genres. To address the issues mentioned above, in this paper, we propose a multi-Task contrastive learning framework for semi-supervised singing melody extraction, termed as MCSSME. Specifically, to deal with data scarcity limitation, we propose a self-consistency regularization (SCR) method to train the model on the unlabeled data. Transformations are applied to the raw signal of polyphonic music, which makes the network to improve its representation capability via recognizing the transformations. We further propose a novel multi-task learning (MTL) approach to jointly learn singing melody extraction and classification of transformed data. To deal with generalization limitation, we also propose a contrastive embedding learning, which strengthens the intra-class compactness and inter-class separability. To improve the generalization on different music genres, we also propose a domain classification method to learn task-dependent features by mapping data from different music genres to shared subspace. MCSSME evaluates on a set of well-known public melody extraction datasets with promising performances. The experimental results demonstrate the effectiveness of the MCSSME framework for singing melody extraction from polyphonic music using very limited labeled data scenarios.

Hot rolled prognostic approach based on hybrid Bayesian progressive layered extraction multi-task learning

Hot-rolled strip products have diverse applications, and enhancing the detection, diagnostics, and prognostics of product quality during hot rolling is essential. Nevertheless, the multivariable, strong coupling, nonlinear, and time-varying nature of the production process poses a rigorous challenge for accurate hot-rolled prognostics. This paper implements a progressive layered extraction (PLE) multi-task learning (MTL) framework to simultaneously estimate multiple quality indicators, such as strip crown, center line deviation, exit temperature, wedge, width, and symmetry flatness. Additionally, the paper proposes the implements of Hybrid Bayesian Neural Network (HBNN) experts and a gating network with attention mechanism to integrate private and shared task features. It also puts forth an auxiliary task involving a Variational Autoencoder with Generative Adversarial Networks (VAE-GAN) to extract latent states from the original sequence. Moreover, an adaptive joint loss optimization is employed to update the weight of individual task losses for MTL training problems, and three sets of field hot-rolled datasets are used for model evaluation. In the experimental validation, considering the noisy field data and limited conditions in the real hot rolled production, comparative experiments are conducted to demonstrate the improved generalization and robustness of the proposed MTL approach. These experiments involve different percentages of the total data, ranging from 5% to 20%, and various prediction horizons ranging from 1 to 50 steps for model establishment. In addition, the paper discusses the interpretation of the model and strategies for further enhancing model performance.

Time series prediction for production quality in a machining system using spatial-temporal multi-task graph learning

The quality of multiple-machining-feature parts in batch computer numerical control machining systems is affected by complex spatial-temporal relationships, making accurate prediction difficult. Traditional quality prediction methods are difficult to cope with the challenges posed by the time impact effects generated by batch production sequences and the spatial impact effects generated by the adjacency of machining features of thin-walled parts. Hence, the time series prediction for production quality in a machining system using spatial-temporal multi-task graph learning is proposed to analyze the spatial-temporal impact relationship of machining quality and achieve accurate quality prediction. Specifically, a graph learning-based framework to capture the coupled spatial-temporal effects is designed to represent the spatial topological and temporal relationship of the machining features of batch parts. A novel global-local multi-task spatial-temporal graph learning framework is constructed to simultaneously predict the time series quality of each machining feature at different spatial positions. The global multi-task model aims to extract spatial-temporal coupling data features (i.e., input parameters for machining features at different positions of batch parts) for machining feature clusters based on gated networks and spatial-temporal graph convolution with multi-head attention. Then, the local multi-task model for each feature cluster (round holes, rectangular grooves, etc.) performs individual time series quality prediction regarding multiple quality indicators. Experimental results on a production dataset of a computer numerical control machining system for thin-walled parts show that the proposed method is superior to the conventional models such as a simple spatial-temporal graph convolution method (3.0%, 18.3% and 11.0% improvement in R2, MAE and RMSE) or the model without graph approach (6.5%, 29.8% and 19.8% improvement in R2, MAE and RMSE).

Improving Vessel Segmentation with Multi-Task Learning and Auxiliary Data Available Only During Model Training

Liver vessel segmentation in magnetic resonance imaging data is important for the computational analysis of vascular remodeling, associated with a wide spectrum of diffuse liver diseases. Existing approaches rely on contrast enhanced imaging data, but the necessary dedicated imaging sequences are not uniformly acquired. Images without contrast enhancement are acquired more frequently, but vessel segmentation is challenging, and requires large-scale annotated data. We propose a multi-task learning framework to segment vessels in liver MRI without contrast. It exploits auxiliary contrast enhanced MRI data available only during training to reduce the need for annotated training examples. Our approach draws on paired native and contrast enhanced data with and without vessel annotations for model training. Results show that auxiliary data improves the accuracy of vessel segmentation, even if they are not available during inference. The advantage is most pronounced if only few annotations are available for training, since the feature representation benefits from the shared task structure. A validation of this approach to augment a model for brain tumor segmentation confirms its benefits across different domains. An auxiliary informative imaging modality can augment expert annotations even if it is only available during training.

Self-supervised multi-task learning framework for safety and health-oriented road environment surveillance based on connected vehicle visual perception

Cutting-edge connected vehicle (CV) technologies have drawn much attention in recent years. The real-time traffic data captured by a CV can be shared with other CVs and data centers so as to open new possibilities for solving diverse transportation problems. The trajectory data of CVs have been well-studied and widely used. However, image data captured by onboard cameras in a connected environment, as being a kind of fundamental data source, are not sufficiently investigated, especially for safety and health-oriented visual perception. In this paper, a bidirectional process of image synthesis and decomposition (BPISD) approach is proposed, and thus a novel self-supervised multi-task learning framework, to simultaneously estimate depth map, atmospheric visibility, airlight, and PM2.5 mass concentration, in which depth map and visibility are considered highly associated with traffic safety, while airlight and PM2.5 mass concentration are directly correlated with human health. Both the training and testing phases of the proposed system solely require a single image as input. Due to the innovative training pipeline, the depth estimation network can automatically manage various levels of visibility conditions and overcome diverse inherent problems in current image-synthesis-based self-supervised depth estimation, thereby generating high-quality depth maps even in low-visibility situations and further benefiting accurate estimations of visibility, airlight, and PM2.5 mass concentration. Extensive experiments on the original and synthesized data from the KITTI dataset and real-world data collected in Beijing demonstrate that the proposed method can (1) achieve performance comparable in self-supervised depth estimation as compared with other state-of-the-art methods when taking clear images as input; (2) predict vivid depth map for images contaminated by various levels of haze when the network trained with previous framework fails; and (3) accurately estimate visibility, airlight, and PM2.5 mass concentrations. Beneficial applications can be developed based on the presented work to contribute to high-precise and dynamic geoinformation reconstruction, transportation, meteorology, and smart city.

PETS-Nets: Joint Pose Estimation and Tissue Segmentation of Fetal Brains Using Anatomy-Guided Networks.

Fetal Magnetic Resonance Imaging (MRI) is challenged by fetal movements and maternal breathing. Although fast MRI sequences allow artifact free acquisition of individual 2D slices, motion frequently occurs in the acquisition of spatially adjacent slices. Motion correction for each slice is thus critical for the reconstruction of 3D fetal brain MRI. In this paper, we propose a novel multi-task learning framework that adopts a coarse-to-fine strategy to jointly learn the pose estimation parameters for motion correction and tissue segmentation map of each slice in fetal MRI. Particularly, we design a regression-based segmentation loss as a deep supervision to learn anatomically more meaningful features for pose estimation and segmentation. In the coarse stage, a U-Net-like network learns the features shared for both tasks. In the refinement stage, to fully utilize the anatomical information, signed distance maps constructed from the coarse segmentation are introduced to guide the feature learning for both tasks. Finally, iterative incorporation of the signed distance maps further improves the performance of both regression and segmentation progressively. Experimental results of cross-validation across two different fetal datasets acquired with different scanners and imaging protocols demonstrate the effectiveness of the proposed method in reducing the pose estimation error and obtaining superior tissue segmentation results simultaneously, compared with state-of-the-art methods.

Efficacy of Regularized Multitask Learning Based on SVM Models.

This article investigates the efficacy of a regularized multitask learning (MTL) framework based on SVM (M-SVM) to answer whether MTL always provides reliable results and how MTL outperforms independent learning. We first find that the M-SVM is Bayes risk consistent in the limit of a large sample size. This implies that despite the task dissimilarities, the M-SVM always produces a reliable decision rule for each task in terms of the misclassification error when the data size is large enough. Furthermore, we find that the task-interaction vanishes as the data size goes to infinity, and the convergence rates of the M-SVM and its single-task counterpart have the same upper bound. The former suggests that the M-SVM cannot improve the limit classifier's performance; based on the latter, we conjecture that the optimal convergence rate is not improved when the task number is fixed. As a novel insight into MTL, our theoretical and experimental results achieved an excellent agreement that the benefit of the MTL methods lies in the improvement of the preconvergence-rate (PCR) factor (to be denoted in Section III) rather than the convergence rate. Moreover, this improvement of PCR factors is more significant when the data size is small. In addition, our experimental results of five other MTL methods demonstrate the generality of this new insight.

A multi-task deep learning framework for forecasting sparse demand of demand responsive transit

Demand Responsive Transport (DRT) is a flexible transportation service that adjusts its routes and schedules in real time based on user requests. Real-time demand forecasting has become a timely issue since the resolution of transportation planning has increased to provide flexible services to users. Although many previous studies have been conducted to design the demand forecasting model using the mobility service historical data, there are more considerations when predicting real-time demand for on-demand mobility services. Therefore, this study aims to develop a demand forecasting model for DRT services addressing two issues. First, repeated calls for service and abandoned requests are converted into the unit of demand to capture served and potential demand. Second, given the sparsity of DRT historical data, a multi-task deep learning model is proposed to predict both the probability and volume of demand simultaneously to mitigate the data sparsity problem. A zero-inflated loss function was introduced to consider the skewed demand distribution. To attain our goals, the historical dataset of the I-MOD service trial in Yeongjongdo, Inchon, Korea was used. The performance of the proposed model is compared against several baseline models, demonstrating its superiority in forecasting demand for mobility services. The results show that a multi-task structure can alleviate the data sparsity problem by excluding where demand is not generated from the forecasting problem. The proposed models can provide insights into real-time demand forecasting with sparse demand.

Enhanced Multitask Learning for Hash Code Generation of Palmprint Biometrics.

This paper presents a novel multitask learning framework for palmprint biometrics, which optimizes classification and hashing branches jointly. The classification branch within our framework facilitates the concurrent execution of three distinct tasks: identity recognition and classification of soft biometrics, encompassing gender and chirality. On the other hand, the hashing branch enables the generation of palmprint hash codes, optimizing for minimal storage as templates and efficient matching. The hashing branch derives the complementary information from these tasks by amalgamating knowledge acquired from the classification branch. This approach leads to superior overall performance compared to individual tasks in isolation. To enhance the effectiveness of multitask learning, two additional modules, an attention mechanism module and a customized gate control module, are introduced. These modules are vital in allocating higher weights to crucial channels and facilitating task-specific expert knowledge integration. Furthermore, an automatic weight adjustment module is incorporated to optimize the learning process further. This module fine-tunes the weights assigned to different tasks, improving performance. Integrating the three modules above has shown promising accuracies across various classification tasks and has notably improved authentication accuracy. The extensive experimental results validate the efficacy of our proposed framework.

Probabilistic load forecasting for integrated energy systems using attentive quantile regression temporal convolutional network

The burgeoning proliferation of integrated energy systems has fostered an unprecedented degree of coupling among various energy streams, thereby elevating the necessity for unified multi-energy forecasting (MEF). Prior approaches predominantly relied on independent predictions for heterogeneous load demands, overlooking the synergy embedded within the dataset. The two principal challenges in MEF are extracting the intricate coupling correlations among diverse loads and accurately capturing the inherent uncertainties associated with each type of load. This study proposes an attentive quantile regression temporal convolutional network (QTCN) as a probabilistic framework for MEF, featuring an end-to-end predictor for the probabilistic intervals of electrical, thermal, and cooling loads. This study leverages an attention layer to extract correlations between diverse loads. Subsequently, a QTCN is implemented to retain the temporal characteristics of load data and gauge the uncertainties and temporal correlations of each load type. The multi-task learning framework is deployed to facilitate simultaneous regression of various quantiles, thereby expediting the training progression of the forecasting model. The proposed model is validated using realistic load data and meteorological data from the Arizona State University metabolic system and National Oceanic and Atmospheric Administration respectively, and the results indicate superior performance and greater economic benefits compared to the baselines in existing literature.

Deep Multi-task Learning for Animal Chest Circumference Estimation from Monocular Images

AbstractThe applications of deep learning algorithms with images to various scenarios have attracted significant research attention. However, application scenarios in animal breeding managements are still limited. In this paper we propose a new deep learning framework to estimate the chest circumference of domestic animals from images. This parameter is a key metric for breeding and monitoring the quality of animal in animal husbandry. We design a set of feature extraction methods based on a multi-task learning framework to address the challenging issues in the main estimation task. The multiple tasks in our proposed framework include object segmentation, keypoint estimation, and depth estimation of cow from monocular images. The domain-specific features extracted from these tasks improve upon our main estimation task. In addition, we also attempt to reduce unnecessary computations during the framework design to reduce the cost of subsequent practical implementation of the developed system. Our proposed framework is tested on our own collected dataset to evaluate its performance.

A Hybrid Convolutional and Recurrent Neural Network for Multi-Sensor Pile Damage Detection with Time Series.

Machine learning (ML) algorithms are increasingly applied to structure health monitoring (SHM) problems. However, their application to pile damage detection (PDD) is hindered by the complexity of the problem. A novel multi-sensor pile damage detection (MSPDD) method is proposed in this paper to extend the application of ML algorithms in the automatic identification of PDD. The time-series signals collected by multiple sensors during the pile integrity test are first processed by the traveling wave decomposition (TWD) theory and are then input into a hybrid one-dimensional (1D) convolutional and recurrent neural network. The hybrid neural network can achieve the automatic multi-task identification of pile damage detection based on the time series of MSPDD results. Finally, the analytical solution-based sample set is utilized to evaluate the performance of the proposed hybrid model. The outputs of the multi-task learning framework can provide a detailed description of the actual pile quality and provide strong support for the classification of pile quality as well.

Multi-task learning using non-linear autoregressive models and recurrent neural networks for tide level forecasting

Tide level forecasting plays an important role in environmental management and development. Current tide level forecasting methods are usually implemented for solving single task problems, that is, a model built based on the tide level data at an individual location is only used to forecast tide level of the same location but is not used for tide forecasting at another location. This study proposes a new method for tide level prediction at multiple locations simultaneously. The method combines nonlinear autoregressive moving average with exogenous inputs (NARMAX) model and recurrent neural networks (RNNs), and incorporates them into a multi-task learning (MTL) framework. Experiments are designed and performed to compare single task learning (STL) and MTL with and without using non-linear autoregressive models. Three different RNN variants, namely, long short-term memory (LSTM), gated recurrent unit (GRU) and bidirectional LSTM (BiLSTM) are employed together with non-linear autoregressive models. A case study on tide level forecasting at many different geographical locations (5 to 11 locations) is conducted. Experimental results demonstrate that the proposed architectures outperform the classical single-task prediction methods.